A common problem that is encountered in training neural networks for prediction, forecasting, pattern recognition, sensor validation and/or processing problems is that some of the training/testing patterns might be missing, corrupted, and/or incomplete. Prior systems merely discarded data with the result that some areas of the input space may not have been covered during training of the neural network. For example, if the network is utilized to learn the behavior of a chemical plant as a function of the historical sensor and control settings, these sensor readings are typically sampled electronically, entered by hand from gauge readings and/or entered by hand from laboratory results. It is a common occurrence that some or all of these readings may be missing at a given time. It is also common that the various values may be sampled on different time intervals. Additionally, any one value may be “bad” in the sense that after the value is entered, it may be determined by some method that a data item was, in fact, incorrect. Hence, if the data were plotted in a table, the result would be a partially filled-in table with intermittent missing data or “holes”, these being reminiscent of the holes in Swiss cheese. These “holes” correspond to “bad” or “missing” data. The “Swiss-cheese” data table described above occurs quite often in real-world problems.
Conventional neural network training and testing methods require complete patterns such that they are required to discard patterns with missing or bad data. The deletion of the bad data in this manner is an inefficient method for training a neural network. For example, suppose that a neural network has ten inputs and ten outputs, and also suppose that one of the inputs or outputs happens to be missing at the desired time for fifty percent or more of the training patterns. Conventional methods would discard these patterns, leading to training for those patterns during the training mode and no reliable predicted output during the run mode. This is inefficient, considering that for this case more than ninety percent of the information is still there for the patterns that conventional methods would discard. The predicted output corresponding to those certain areas will be somewhat ambiguous and erroneous. In some situations, there may be as much as a 50% reduction in the overall data after screening bad or missing data. Additionally, experimental results have shown that neural network testing performance generally increases with more training data, such that throwing away bad or incomplete data decreases the overall performance of the neural network.
In addition to the above, when data is retrieved on different time scales, it is necessary to place all of the data on a common time scale. However, this is difficult in that for a given time scale, another and longer time scale results in missing data at that position. For example, if one set of data were taken on an hourly basis and another set of data were taken on a quarter hour basis, there would be three areas of missing data if the input time scale is fifteen minutes. This data must be filled in to assure that all data is presented at synchronized times to the system model. Worse yet, the data sample periods may be non-periodic, producing totally asynchronous data.
In addition, this data may be taken on different machines in different locations with different operating systems and quite different data formats. It is essential to be able to read all of these different data formats, keeping track of the data value and the time-stamp of the data out to one or more “flat files” which are column oriented, each column corresponding to a data variable and/or the data/time stamp of the variable. It is a formidable task to retrieve this data keeping track of the date-time information and read it into an internal data-table (spreadsheet) so that the data can be time merged.
Another aspect of data integrity is that with respect to inherent delays in a system. For example, in a chemical processing system, a flow meter output can provide data at time t0 at a given value. However, a given change in flow resulting in a different reading on the flow meter may not affect the output for a predetermined delay τ. In order to predict what the output would be, this flow meter output must be input to the network at a delay equal to τ. This must also be accounted for in the training of the network. In generating data that accounts for time delays, it has been postulated that it would be possible to generate a table of data that comprises both original data and delayed data. This necessitates a significant amount of storage in order to store all of the delayed data and all of the original data, wherein only the delayed data is utilized. Further, in order to change the value of the delay, an entirely new set of input data must be generated off the original set.